Skip to content

Specify EntityLocation alignment to improve codegen - #26042

Merged
alice-i-cecile merged 2 commits into
bevyengine:mainfrom
dylansechet:fix_unsafe_world_cell_regression
Oct 7, 2026
Merged

alice-i-cecile merged 2 commits into
bevyengine:mainfrom
dylansechet:fix_unsafe_world_cell_regression

Conversation

@dylansechet

@dylansechet dylansechet commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Objective

Fixes #25839.

Solution

#25632 changed the memory layout of UnsafeEntityCell, which caused some codegen downstream to hit a store-to-load forwarding failure. (I hadn't heard about those before working on this PR, this blogpost has a nice explainer. TLDR is that it's slow).

Adding a repr(align(8)) to EntityLocation seems to tip LLVM codegen the right way and we get to avoid that store-to-load forwarding failure, which lets us recover the pre-#25632 performance.

The PR also speeds up World::entity, which already had a similar forwarding failure before #25632.

I want all the details!

EntityLocation gets loaded from the entity table into three registers: archetype_row (4 bytes), table_row+archetype_id (8 bytes), and table_id (4 bytes). Since #25632, it then gets written as 8+4+4 into UnsafeEntityCell: archetype_row+table_row, then archetype_id, then table_id. That first 8-byte chunk needs archetype_row plus the half of the second register, and LLVM builds it by writing both registers to the stack and reading 8 bytes back. That read spans two pending stores and causes the store-to-load forwarding to fail. This path gets hit in most places where we build an UnsafeEntityCell from a location, and so ends up affecting multiple benchmarks.

Putting a repr(align(8)) onto EntityLocation makes LLVM load it as archetype_row, table_row, and archetype_id+table_id (4+4+8), and it can then join archetype_row and table_row directly in a register without the round-trip to the stack.

Testing

Ran the benchmarks mentioned in #25839.

We get a bunch of nice improvements, with observe/observer_custom/10000_entity being the only one that's still worse than before #25632. Not sure what's going on there, and will probably leave it as future work.

benchmark Before #25632 #25632 #25632 + align(8)
world_entity/50000_entities 259.52 µs (x1.00) 233.33 µs (x0.90) 66.48 µs (x0.26)
event_propagation/four_event_types 296.68 µs (x1.00) 350.13 µs (x1.18) 310.61 µs (x1.05)
observe/observer_custom/10000_entity 461.24 µs (x1.00) 513.55 µs (x1.11) 510.81 µs (x1.11)
ecs::resources::insert_remove 75.7 ns (x1.00) 76.7 ns (x1.01) 76.9 ns (x1.02)
despawn_world_recursive/1_entities 297.8 ns (x1.00) 315.6 ns (x1.06) 305.4 ns (x1.03)
despawn_world_recursive/100_entities 12.72 µs (x1.00) 12.90 µs (x1.01) 12.49 µs (x0.98)
despawn_world_recursive/10000_entities 1.23 ms (x1.00) 1.25 ms (x1.02) 1.21 ms (x0.99)
query_get_many_2/50000_calls_table 410.63 µs (x1.00) 412.17 µs (x1.00) 401.98 µs (x0.98)
query_get_many_2/50000_calls_sparse 307.48 µs (x1.00) 308.31 µs (x1.00) 291.51 µs (x0.95)
query_get_many_5/50000_calls_table 960.35 µs (x1.00) 957.31 µs (x1.00) 974.13 µs (x1.01)
query_get_many_5/50000_calls_sparse 715.22 µs (x1.00) 714.31 µs (x1.00) 679.33 µs (x0.95)
query_get_many_10/50000_calls_table 1.75 ms (x1.00) 1.75 ms (x1.00) 1.76 ms (x1.01)
query_get_many_10/50000_calls_sparse 1.63 ms (x1.00) 1.62 ms (x1.00) 1.59 ms (x0.98)
get_entity_mut_slice/size/20 102.2 ns (x1.00) 152.1 ns (x1.49) 100.9 ns (x0.99)
get_entity_mut_slice/size/200 1.96 µs (x1.00) 2.43 µs (x1.24) 1.90 µs (x0.97)
get_entity_mut_slice/size/2000 18.95 µs (x1.00) 24.44 µs (x1.29) 18.95 µs (x1.00)

I started by pointing a LLM at the issue and asked it to inspect the assembly before and after a716a99 to try and find a cause. It directed me towards the store-to-load issue. I'd never heard of that so I did a bunch of reading to understand what was going on, then spent way too much time staring at the assembly, and finally ran the benchmarks.

@alice-i-cecile alice-i-cecile added A-ECS Entities, components, systems, and events C-Performance A change motivated by improving speed, memory usage or compile times D-Straightforward Simple bug fixes and API improvements, docs, test and examples S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Oct 6, 2026
@alice-i-cecile alice-i-cecile added this to the 0.20 milestone Oct 6, 2026

@hymm hymm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it was a fun hole to jump down.

@alice-i-cecile alice-i-cecile added S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Oct 6, 2026
@alice-i-cecile
alice-i-cecile added this pull request to the merge queue Oct 7, 2026
Merged via the queue into bevyengine:main with commit a144d28 Oct 7, 2026
73 of 75 checks passed
mockersf pushed a commit that referenced this pull request Oct 8, 2026
# Objective

Fixes #25839.

## Solution

#25632 changed the memory layout of `UnsafeEntityCell`, which caused
some codegen downstream to hit a store-to-load forwarding failure. (I
hadn't heard about those before working on this PR, [this
blogpost](https://eme64.github.io/blog/2024/06/24/Auto-Vectorization-and-Store-to-Load-Forwarding.html)
has a nice explainer. TLDR is that it's slow).

Adding a `repr(align(8))` to `EntityLocation` seems to tip LLVM codegen
the right way and we get to avoid that store-to-load forwarding failure,
which lets us recover the pre-#25632 performance.

The PR also speeds up `World::entity`, which already had a similar
forwarding failure before #25632.

<details>
<summary>I want all the details!</summary>

`EntityLocation` gets loaded from the entity table into three registers:
`archetype_row` (4 bytes), `table_row`+`archetype_id` (8 bytes), and
`table_id` (4 bytes). Since #25632, it then gets written as 8+4+4 into
`UnsafeEntityCell`: `archetype_row`+`table_row`, then `archetype_id`,
then `table_id`. That first 8-byte chunk needs `archetype_row` plus the
half of the second register, and LLVM builds it by writing both
registers to the stack and reading 8 bytes back. That read spans two
pending stores and causes the store-to-load forwarding to fail. This
path gets hit in most places where we build an `UnsafeEntityCell` from a
location, and so ends up affecting multiple benchmarks.

Putting a `repr(align(8))` onto `EntityLocation` makes LLVM load it as
`archetype_row`, `table_row`, and `archetype_id`+`table_id` (4+4+8), and
it can then join `archetype_row` and `table_row` directly in a register
without the round-trip to the stack.

</details>



## Testing
Ran the benchmarks mentioned in #25839. 

We get a bunch of nice improvements, with
`observe/observer_custom/10000_entity` being the only one that's still
worse than before #25632. Not sure what's going on there, and will
probably leave it as future work.

| benchmark | Before #25632 | #25632 | #25632 + align(8) |
|---|---|---|---|
| `world_entity/50000_entities` | 259.52 µs (x1.00) | 233.33 µs (x0.90)
| 66.48 µs (x0.26) |
| `event_propagation/four_event_types` | 296.68 µs (x1.00) | 350.13 µs
(x1.18) | 310.61 µs (x1.05) |
| `observe/observer_custom/10000_entity` | 461.24 µs (x1.00) | 513.55 µs
(x1.11) | 510.81 µs (x1.11) |
| `ecs::resources::insert_remove` | 75.7 ns (x1.00) | 76.7 ns (x1.01) |
76.9 ns (x1.02) |
| `despawn_world_recursive/1_entities` | 297.8 ns (x1.00) | 315.6 ns
(x1.06) | 305.4 ns (x1.03) |
| `despawn_world_recursive/100_entities` | 12.72 µs (x1.00) | 12.90 µs
(x1.01) | 12.49 µs (x0.98) |
| `despawn_world_recursive/10000_entities` | 1.23 ms (x1.00) | 1.25 ms
(x1.02) | 1.21 ms (x0.99) |
| `query_get_many_2/50000_calls_table` | 410.63 µs (x1.00) | 412.17 µs
(x1.00) | 401.98 µs (x0.98) |
| `query_get_many_2/50000_calls_sparse` | 307.48 µs (x1.00) | 308.31 µs
(x1.00) | 291.51 µs (x0.95) |
| `query_get_many_5/50000_calls_table` | 960.35 µs (x1.00) | 957.31 µs
(x1.00) | 974.13 µs (x1.01) |
| `query_get_many_5/50000_calls_sparse` | 715.22 µs (x1.00) | 714.31 µs
(x1.00) | 679.33 µs (x0.95) |
| `query_get_many_10/50000_calls_table` | 1.75 ms (x1.00) | 1.75 ms
(x1.00) | 1.76 ms (x1.01) |
| `query_get_many_10/50000_calls_sparse` | 1.63 ms (x1.00) | 1.62 ms
(x1.00) | 1.59 ms (x0.98) |
| `get_entity_mut_slice/size/20` | 102.2 ns (x1.00) | 152.1 ns (x1.49) |
100.9 ns (x0.99) |
| `get_entity_mut_slice/size/200` | 1.96 µs (x1.00) | 2.43 µs (x1.24) |
1.90 µs (x0.97) |
| `get_entity_mut_slice/size/2000` | 18.95 µs (x1.00) | 24.44 µs (x1.29)
| 18.95 µs (x1.00) |


---

I started by pointing a LLM at the issue and asked it to inspect the
assembly before and after a716a99 to try and find a cause. It directed
me towards the store-to-load issue. I'd never heard of that so I did a
bunch of reading to understand what was going on, then spent way too
much time staring at the assembly, and finally ran the benchmarks.

Co-authored-by: Alice Cecile <alice.i.cecile@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ECS Entities, components, systems, and events C-Performance A change motivated by improving speed, memory usage or compile times D-Straightforward Simple bug fixes and API improvements, docs, test and examples S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Pref Regressions for "Use NonNull in UnsafeWorldCell (#25632)"

3 participants